Skip to main content

Transfer learning & Prediction

Summary​

  • In order to predict the defects on ebmud videos, we used transfer learning by utilizing existing models trained on the utilities like DNV and SD1.
  • The details on models used, data used and observations are all docuemnted in a excel sheet, currently available on Srujana's Onedrive here: OneDrive > Documents > cctv-ebmud-transfer-learning.xlsx
  • Sudhir reviewed testing various utility models with ebmud and the above excel sheet.

Models​

  • The models are present on GDrive here: /VS_Research/CCTV/models/
  • The naming convention of models, provided by Vannary, is here:
      1. Any model name with "cctv-multilabel*" is a one-stage model.
      2. Any model name with "cctv-singlelabel*" is a binary model that I use to predict defective vs non-defective images. This model is the first model for the two-stage model approach.
      3. Any model name with "cctv-defect-multilabel*" is the model without ND (no defect) in the prediction class. I use this model as the second model for the two-stage model approach after running the "cctv-singlelabel*" model to label images with the type of defect/feature.
      4. Any model name with "*spvidT*" is a model that is trained to test with the VB dataset. However, any model is acceptable if you plan to use the model for transfer learning and test the model on different test datasets.
      5. The naming convention is as follows:
          * Type of model -  utility - train data set - C (include continuous labels) - Extra info (i.e. spvidT (for VB), GL for Group Label) - train/test split -  cross-validation set that I used for training - backbone model - pretrain/w&b setting (T for True, F for False) -  number of epoch - number of freezing epoch - image augmentation (T for true and F for false) - weighted function (T for true and F for False) - number of gpus used to train the model *
      6. A one-stage ResNet50 model for DNV:
        - for FB: cctv-multilabel-dnv-791_v3-C-80_20-cv0-resnet50-prT-e10-fe10-augF-weightF-4gpus
        - for VB: cctv-multilabel-dnv-791_v3-C-spvidT-80_20-cv0-resnet50-prT-e10-fe10-augF-weightF-4gpus
        - You can use any of these two models if you are not planning to test on the DNV dataset. I suggest using FB model one because it was trained with images from all videos.

Procedure​

Steps taken to test transfer-learning for ebmud on cctv-apps are below. The first section is old CCTV usage piepline and next section contains latest CCTV usage pipeline. The difference between them is the creation of metadata files, which are optional in new approach.

  • If you are working completely on MSI, follow steps in the
    • Old section except for the Metadata creation steps, or
    • New section skipping scp step.
  • If you are working with 2 SSH machine , like microcenter and MSI, you can follow new approach.

CCTV usage - Old​

  1. Download the videos from GDrive at MyDrive/CCTV/ebmud/ to MSI at /home/gqc/CCTV/ebmud/Data/Dataset_X/Received_Data.
  2. Download required models from GDrive at MyDrive/VS_Research/CCTV/models/ to MSI at /home/gqc/git/cctv-apps/models using rclone like this:
      (pytorch) gqc@msi-desktop-ubuntu:models$ rclone copy hydrotrek-gdrive:/VS_Research/CCTV/models/DNV/cctv-multilabel-dnv-791_v3-C-80_20-cv0-resnet50-prT-e10-fe4-augF-weightF-4gpus.pkl .
  3. Edit the YAML parameters.
  4. (Optional) Prepare distance annotation and defect annotation csv files under /home/gqc/CCTV/ebmud/Data/Dataset_X/Metadata_CSV
    • Work needs to be done to save the annotation data to DB and link that DB here instead of using CSV files.
    • Having 'None' for metedata is throwing errors currently (This has been fixed).
  5. Refer gqc-utility-notebooks/nbs/01_CCTV Usage/02-usage-running_cctv_image_processing_pipeline.ipynb for commands to run.
       cctv_create_video_lists -y <path_to_cctv_settings.yml>  # To form video groups
       cctv_extract_frames -y <path_to_cctv_settings.yml>  # To extract frames
  6. Copy the zipped frames to cctv-apps/public and extract frames here.
  7. Create CSV in cctv-apps for the extracted frames. We will use this later to upload in the app. Also, if working on MSI, make sure to download them to local.
  8. Run the streamlit app. On AI-Prediction app, Select the model to use for prediction from dropdown, labels present or not, annotations option and upload the CSV we created before.
  9. Select images from the carousal to generate prediction for that image.

CCTV usage - New​

The below steps involves 2 machines: Microcenter Ubuntu machine and MSI Ubuntu. The difference arrives only during file transfer to work with models.

  1. Create a datset folder such that hierarchy is similar to this: /cctv/ebmud/Data/Dataset_100923/.
  2. Copy the videos from External drive or Google drive (refer above) to /ebmud/Data/dataset_100923/Received_Data/ and organize them in sub-folders to be grouped later.
  3. Edit the YAML parameters along with EXTRACT_N_TH_FRAME which defines the number of frames to be extracted per second. For this dataset, the videos have 60 fps so we picked EXTRACT_N_TH_FRAME = 30 to extract 2 frames per second.
  4. Refer gqc-utility-notebooks/nbs/01_CCTV Usage/02-usage-running_cctv_image_processing_pipeline.ipynb for commands to run.
       cctv_create_video_lists -y <path_to_cctv_settings.yml>  # To form video groups
       cctv_extract_frames -y <path_to_cctv_settings.yml>  # To extract frames
  5. Since we want to test transfer learning on MSI, we use below command to transfer files from Microcenter to MSI over SSH. In the microcenter terminal, for example, use below command to transfer the dataset to MSI destination folder:
     scp -r ebmud/Data/Dataset_100923 gqc@192.168.50.235:/home/gqc/CCTV/ebmud/Data/
  6. Copy the zipped frames from Extracted_Frames to cctv-apps/public and extract frames here using unzip.
  7. Create CSV in cctv-apps for the extracted frames.
  8. Run the streamlit app. On AI-Prediction app, Select the model to use for prediction from dropdown, labels present or not (check CSV for labels column), annotations option and upload the CSV we created before.
  9. Select images from the carousal to generate prediction for that image.
Hack
  • Given n set of groups, assume a new group or groups are added. To run all the steps above afor new data along with old data is a waste of resources.
  • One quick way to not process old groups (considering there are no changes in them), is by listing the list of groups to ignore in the yaml configuration variable VIDEO_GROUPS_TO_SKIP